Summary

• port the challenge harness to the Swift runtime path and keep trusted benchmark/correctness logic outside editable submission files
• keep hidden benchmark oracle tokens out of the runtime worker by using begin/step benchmark decode requests
• run correctness and timed benchmark in isolated runtime worker processes with private prompt/golden env stripped from the worker
• load private benchmark golden and hidden GPQA reference gates from R2, while keeping the public local correctness gate in-repo
• capture hidden GPQA TTFT during the existing first-token behavior correctness gate, so there is no duplicate hidden-prompt model pass
• add a private Claude semantic GPQA diagnostic that records only aggregate pass counts in score.json
• harden transform and benchmark execution with offline sandboxing, editable-path overlay validation, artifact redaction, and private-artifact deny checks
• batch the public 256-token teacher-forced correctness gate inside the worker so the gate stays strong without 256 single-token worker calls
• remove hardware bandwidth measurement from the benchmark path; primary ranking is prefill/decode speedup only, with expert-streaming byte counters kept as diagnostics
• score ranked submissions by official-run prefill/decode speedup, with per-component speedup floors so a submission cannot trade a large regression in one path for a gain in the other

Gates and Checks

The model-behavior checks are intentionally layered. Each one catches a different way a submission could be fast but no longer behave like the reference model.

1. Correctness Check

• Mode: hard gate.
• What it checks: teacher-forced token correctness on a 512-token prompt with a 256-token expected continuation. Local runs use the public checked-in fixture; official benchmark runs use the hidden R2 golden.
• Why we do it: this is the first proof that the changed runtime/kernels still compute the same model path before we accept timing numbers.
• What it prevents: broken tokenizer/model loading, wrong logits, routing/cache regressions, arbitrary fast token output, and submissions that only work on the public local prompt.

2. Hidden GPQA Single-Token Check

• Mode: hard gate.
• What it checks: hidden GPQA-style science QA prompts must produce an accepted first answer token. The same pass records aggregate TTFT stats.
• Why we do it: the long correctness prompt alone is too narrow. A hidden first-token QA check forces the model to preserve short-answer behavior on a different prompt style while keeping runtime cost manageable.
• What it prevents: optimizing only for the transcription-style correctness prompt, first-token answer corruption, prompt-specific hacks, and fake decode paths that do not preserve basic QA behavior.

3. Hidden GPQA Multi-Token Check

• Mode: diagnostic today; used as input to the semantic check.
• What it checks: the runner generates short hidden GPQA continuations, currently up to 10 new tokens per case, and keeps those candidate answers private.
• Why we do it: a model can pass one-token checks but drift immediately after it consumes its own generated tokens. Short continuations give us a stronger behavior signal without running full long-form QA.
• What it prevents: first-token-only special casing, decode paths that collapse after the first token, and changes that preserve exact token gates while damaging short answer quality.

4. Claude Semantic GPQA Check

• Mode: diagnostic by default; can become a hard gate with MLXFAST_SEMANTIC_GPQA_REQUIRED=1 after calibration.
• What it checks: Claude compares the private GPQA candidate answers against private references and patches only aggregate pass fields into score.json.
• Why we do it: exact token checks are brittle across Apple GPU/MLX differences and can miss semantically wrong but token-near outputs. A judge gives us a behavior signal based on meaning, not just exact token IDs.
• What it prevents: submissions that pass exact-token gates while giving wrong short-answer content, answer-format hacks, and regressions that are visible semantically but not in the first token.
• Privacy: prompts, references, candidate answers, token IDs, accepted sets, and raw judge text stay in the private runner directory and are not uploaded.

Scoring

• Primary score is baseline-normalized timing speedup:
  • decode_speedup = baseline_decode_sec_per_token / decode_sec_per_token
  • prefill_speedup = baseline_prefill_sec_per_token / prefill_sec_per_token
  • score = decode_speedup^0.75 * prefill_speedup^0.25
• Each primary component must also pass a floor:
  • decode_speedup >= 0.95
  • prefill_speedup >= 0.95
• The floor prevents benchmark-maxing one component while materially regressing the other. The weighted score still ranks passing submissions.
• The current baseline constants come from the official Blacksmith run for the active 512-token prefill / 256-token decode benchmark oracle.
• score.json still reports peak RAM, expert read bytes, expert read seconds, cache counters, derived expert-streaming bytes per token, GPQA TTFT aggregates, and semantic GPQA aggregate fields for diagnostics and future guardrails, but only prefill/decode speedups drive the primary score.

Latest Status

• Current PR head: 06da143.
• Latest full Blacksmith benchmark: https://github.com/Layr-Labs/mlxfast-challenge-dev/actions/runs/28281015024
  • Workflow passed end to end in 50m03s.
  • Core benchmark step passed in 28m26s.
  • Hidden correctness plus GPQA first-token gate passed: 265 checked steps across 10 cases.
  • Hidden GPQA TTFT gate passed: 9/9 cases.
  • Score: 1.0127854432595405.
  • Prefill: 0.13911011108398438 sec/token, speedup 1.0187907394591718, floor 0.95, passed.
  • Decode: 2.986097291347657 sec/token, speedup 1.010791554504655, floor 0.95, passed.
  • Semantic GPQA is currently diagnostic and recorded 4/9, so it did not block this run.
• Commit 06da143 adds per-component prefill/decode speedup floors to runtime scoring, artifact validation, tests, README, and challenge docs.
• PR CI on 06da143 passed.

Timing From Latest Run

• Setup/reference checkpoint: 6m35s.
• Transform reference checkpoint: 4s.
• Prepare correctness golden: 2s.
• Core benchmark step: 28m26s.
• Semantic GPQA answer generation: 14m11s.
• Semantic GPQA gate: 37s.

Validation

• swift test --filter 'Score|BenchmarkScript'
• swift test --filter 'Score|BenchmarkScript|BenchmarkSupport'
• swift build -c release
• bash -n .github/scripts/validate-benchmark-artifacts.sh
• bash -n .github/scripts/run-semantic-gpqa-gate.sh
• bash -n benchmark.sh
• bash -n setup.sh
• git diff --check
• Latest full Blacksmith benchmark run passed: https://github.com/Layr-Labs/mlxfast-challenge-dev/actions/runs/28281015024